Enterprises have a requirement for reliable storage of their data. Some enterprises rely on periodic (e.g. nightly) back-ups of their data, with a copy of the data being stored locally or off-site. Other enterprises, particularly financial institutions, have a more stringent requirement for reliable data storage and employ a network which has a primary storage system, for storing data, and a second storage system at a location which is geographically remote from the primary storage system. The storage systems regularly communicate with one another across the network, with the second storage system maintaining an exact copy of the data that is stored on the primary storage system. This process of copying data to the second storage system is often called data mirroring. Usually, the distance between the primary and second storage systems is sufficiently great to survive a typical ‘disaster scenario’, often 50 km or more.
There are several known ways of achieving data mirroring. A first way, known as synchronous mirroring, ensures that the data on the second storage system is, at all times, exactly in step with the data on the primary storage system. This is best described with reference to FIG. 1. A computer system 10 is connected to a primary storage system 11 and, by way of a communication link 12, to a second storage system 13. An instruction to write data (A) is sent to the primary storage system 11. The primary storage system 11 writes the data (B) to a storage device 14 and sends an instruction to write the same data (C) over link 12 to the second storage system 13. As soon as the instruction to write is received, or upon completion of the write operation (D) at the second storage system, an acknowledgement signal (E) is sent from the second storage system 13 to the primary storage system 11, which is returned (F) to the computer system 10. No further instructions to write data are issued by computer system 10 until the acknowledgement signal (F) is received. While this ensures that the primary and second storage systems are always exactly in step with one another, the need to wait for an acknowledgement on each occasion has the disadvantage that the throughput of data can be slow. This is particularly so where the primary and second storage systems are separated by a great distance, e.g. 5000 km, as the propagation of signals across communication link 12 incurs a significant delay. Enterprises which have business sites in different parts of a country, or in different countries or continents, may have the primary storage system at one of their sites and the second storage system at a second of their sites. The delay associated with waiting for acknowledgements is known as latency.
In view of the problems with synchronous mirroring, in a second way of data mirroring, known as asynchronous mirroring, the primary and second storage devices are allowed to be out of step with one another. Referring again to FIG. 1, the computer system 10 sends a write instruction to the primary storage system 11 but does not wait for an acknowledgement from the second storage system 13 before issuing the next write instruction. While this has the advantage of achieving a higher throughput of data, many financial institutions are understandably nervous that operating in this manner could result in them missing details of a vital transaction. If the primary storage system were to fail, the second storage system would be left in an inconsistent and unknown state, potentially costing the institution significant amounts of money, time and reputation.
A method of providing exceptionally reliable transmission of data is known as hitless switching (Described in co-pending U.S. application Ser. Nos. 09/862,864 and 10/154,173, incorporated herein by reference). Hitless switching can be used to, effectively, guarantee the arrival of data at a reception end of an optical transmission system. This can be utilised to extend the reach of synchronous mirroring systems by removing the time delays associated with the transport of acknowledgements over very long distances. This is described in co-pending U.S. application Ser. No. 10/460,285, incorporated herein by reference.
FIG. 2 shows a network utilising hitless switching to provide remote synchronous optical mirroring.
The primary storage system 11 and second storage system 13 are connected to one another by a communication link 20, 21. Transmission equipment 22 connects the primary storage system 11 to the near end of the communication link 20, 21 and transmission equipment 23 connects the second storage system 13 to the far end of the communication link 20, 21. The communication link comprises a first transmission link 20 and a second transmission link 21. The first and second transmission links 20, 21 are routed along physically diverse paths to provide protection against failures in transmission equipment or cables along one of the paths. One possible failure condition is that transmission cables can be accidentally or maliciously cut. The length X of the links can be a considerable distance, such as 5000 km or more. In use, the same data is transmitted over both of the first and second links 20, 21. While the transmission paths 20, 21 are shown as simple direct paths they will, in reality, comprise a series of connections between network nodes such as multiplexers, cross-connects, signal regenerators and other signal processing or switching apparatus. Due to the different routing that paths 20, 21 take, one of the paths will almost certainly be longer than the other.
The control apparatus 24 which implements the function of copying data from the primary storage system 11 to the secondary storage system 13 resides on the primary storage system 11. It receives requests to write data from the host 10, issues write instructions to the second storage system 13 and returns acknowledgements to the host 10 when certain conditions are met.
FIG. 3 shows a form of the transmission apparatus 22 which can be used at the first end of the communication link 20, 21. It comprises a storage buffer 30 for holding a write instruction, transmission formatting equipment 31, 32 for converting the write instruction into an appropriate format for transmission over the links 20, 21 and a control apparatus 33. In the case of an SDH/SONET transmission link, the formatting equipment will format the write instruction into an SDH frame and generate a signal with an appropriate linecode and physical format for transmission over the links 20, 21. In an SDH network, traffic is normally carried in a Synchronous Transmission Module (STM). An STM can support one or more transmission paths depending on the capacity of the STM (STM-1, STM-4, STM-16 and STM-64) and on the data signal carried by the path. The data signal can carry data in data structures, or Virtual Containers (VCs), of differing sizes, such as the VC-4 or VC-12. Each path has an associated path overhead (POH) which enables the network equipment to monitor the transmission of the data signal across the whole path and to perform, for example, quality and error checks. Similarly, for a packet or cell-based transmission format, the formatting equipment 31, 32 will load the write instructions into the payload of a packet and insert appropriate data into the header fields of the packet so as to route the packets across the transmission links 20, 21. These details are well-known to one of ordinary skill in the art. A control apparatus 33 supervises the operation of the equipment. Each write instruction sent by the primary storage system 11 is received by the control apparatus 33 and then inserted into the buffer 30.
The operation of the network will now be described with reference to FIGS. 2 and 3. In operation, the primary host 10 transmits a write request (A) over link 25 to the primary storage system 11, i.e. a request to write a block of data. The control apparatus 24 performs the write operation (B) to the storage devices 14 and, to achieve data mirroring, sends a write request (C), and the appropriate block of data which is to be written, to the transmission equipment 22. This combination of a write request and a block of data to be written will hereafter be called a write instruction. It should be noted that there may be a considerable quantity of data accompanying the write request. The write instruction is held in the transmit buffer 30 of transmission equipment 22 before being launched (D) onto each of the pair of transmission links 20, 21. Control apparatus 33 monitors the status of the buffer 30 and returns an acknowledgement signal (E) to the control apparatus 24 of the primary storage system 11 when the write instruction has been sent from the buffer 30. In this manner, the primary storage system 11, and the host 10, receive an acknowledgement when the write instruction has been sent onto the transmission links 20, 21. Since the transmission over links 20, 21 can be assumed to be secure, the processor 26 in the host 10 can proceed to the next write request.
At the far end transmission equipment 23, two versions of the write instruction will be received, offset from one another by a time period which is due to the different lengths of the transmission paths 20, 21 and the network equipment located along those paths. The two received signals are aligned with one another so that the transmission equipment can select between them without incurring a ‘hit’ on the content, i.e. without duplication or omission of parts of the signal. This is described in co-pending U.S. application Ser. Nos. 09/862,864 and 10/154,173 incorporated herein by reference). Thus, transmission equipment 23 outputs (G) a write instruction constructed from the two received signals. Control apparatus 27 receives the write request and performs a write operation to the storage devices 28.
Hitless switching provides a method of delivering data to a remote point in a highly reliable fashion. However, there is no guarantee or acknowledgement provided that the data has been correctly written to the remote storage device. This is clearly highly undesirable in a remote mirroring situation, in which a guarantee that data has been correctly stored is required.